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ABSTRACT 

Built  around  two  infrared  cameras  and  a  high-speed  opto-mechanical  pointing  mechanism,  the  Infrared 
Eye  attempts  to  mimic  the  human  eye  by  superimposing  a  high-resolution  narrow  field  of  view  (first 
camera)  for  target  identification  over  a  lower  resolution  wide  field  of  view  (second  camera)  for  situation 
awareness.  A  high-performance  opto-mechanical  pointing  system  allows  the  narrow  field  to  be  positioned 
anywhere  within  the  wider  field  providing  the  operator  with  fast  access  to  points  of  interest  without  losing 
situation  awareness. 

During  the  development  of  the  Infrared  Eye,  we  found  that  we  could  use  the  opto-mechanical  pointing 
system  to  produce  high-resolution,  wide  field  of  view  images  at  very  high  speed.  The  pointing  system 
rapidly  moves  the  narrow  field  of  view  to  take  high-resolution  images  that  can  then  be  assembled  into  a 
mosaic  in  a  dynamic  fashion,  taking  into  account  the  movements  of  the  platform  to  optimize  the  selection 
of  mosaic  sub-images.  By  using  the  high-resolution  narrow  field  images  to  fill  the  mosaic,  a  wide  field  of 
view  can  be  covered  with  the  same  high  resolution  as  the  conventional  narrow  field  images. 

As  compared  to  the  current  Infrared  Eye  concept,  the  step-stare  approach  eliminates  one  of  the  two 
cameras  for  a  lighter  and  more  compact  system,  while  increasing  the  resolution  in  the  wide  field  of  view 
coverage,  which  would  be  impossible  using  conventional  cameras  and  optics  as  wide  field  and  high 
resolution  are  mutually  opposed  trade-off  parameters  in  an  optical  design.  The  wide  field  and  high 
resolution  obtained  by  the  step-stare  technique  also  brings  the  detection  capability  in  the  wide  field  to  the 
same  level  as  that  obtained  with  a  narrow  field  camera  in  a  conventional  FLIR  system.  Within  its  related 
R&D  activities,  DRDC  Valcartier  has  demonstrated  the  possibility  to  fabricating  wideband  Risley  prisms 
in  both  the  MWIR  and  LWIR  spectra,  as  well  as  narrower  band  prisms  in  the  visible  and  near  IR  spectra 
for  passive  and  active  imaging.  The  applications  considered  are  UAV  surveillance  platforms  for 
peacekeeping  and  tactical  information  acquisition,  sensor  packages  for  other  manned  airborne  platforms 
or  land-based  perimeter  surveillance.  The  capacity  of  the  Risley  prisms  to  point  the  field  of  view  of  the 
camera  anywhere  within  its  design  coverage  (>  50°)  allows  one  to  configure  the  vertical  and  horizontal 
dimensions  of  the  mosaic  image  according  to  the  requirements  of  the  mission.  A  miniature  prism  package 
is  also  under  design  and  will  be  fitted  in  front  of  small  low-cost,  low-power  micro  IR  imagers  of  the 
160x120  pixels  class.  This  combination  could  provide  mini  UAVs  with  a  high-resolution  IR  payload  under 
3  Kg. 

In  this  paper,  we  will  describe  the  opto-mechanical  pointing  system  used  for  the  step-staring  system  and 
its  different  variations.  We  will  also  describe  the  image  acquisition  and  control  algorithms  developed  to 
build  the  mosaic  dynamically  by  taking  into  account  the  attitude  and  position  of  the  airborne  platform.  A 
simulator  has  been  developed  and  will  be  demonstrated  to  show  how  it  is  used  to  validate  our  image 
acquisition  scenario.  Finally,  imagery  from  our  step-stare  system  will  be  presented  and  discussed. 
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ORGANIZATION 


1.0  INTRODUCTION 

With  a  changing  climate  and  varied  terrain  of  a  15,540,000  square  kilometres  surface  (Canada’s  landmass, 
territorial  waters  and  mid-ocean  sections  of  the  Atlantic  and  Pacific),  search  and  rescue  and  surveillance 
operations  are  a  great  challenge  in  Canada.  These  operations  are  often  carried  out  at  night  or  under  other 
conditions  of  poor  visibility  such  as  smoke,  fog  or  snow.  Many  difficulties  in  surveillance  material 
conception  for  those  operations  arise  from  the  operational  context  of  the  missions.  The  most  significant 
requirement  for  the  system  is  its  capacity  to  allow  the  detection  and  the  identification  of  the  object  of 
interest.  Detection  requires  a  wide  field  of  view,  whereas  identification  requires  a  high  resolution.  This 
presents  a  great  challenge  because  these  concepts  are  opposed  and  it  is  generally  necessary  to  reach  an 
agreement  between  the  two.  One  of  the  solutions  usually  proposed  to  solve  the  resolution  versus  field  of 
view  dilemma  is  to  alternate  between  two  selectable  fields:  a  low  resolution  wide  field  and  a  high 
resolution  narrow  field  at  the  cost  of  loosing  situation  awareness  when  operating  in  narrow  field  of  view 
(NFOV)  and  the  identification  capability  when  using  the  wide  field  of  view. 

An  innovative  approach  was  developed  with  the  Infrared  Eye  project  [1,  2]  at  Defence  R&D  Canada 
Valcartier.  It  tries  to  imitate  the  human  eye  by  superimposing  a  high-resolution  narrow  field  of  view  for 
target  identification  over  a  low-resolution  wide  field  of  view  for  situation  awareness.  A  high  performance 
opto-mechanical  pointing  system  allows  the  narrow  field  to  be  positioned  anywhere  within  the  wider  field. 
The  position  of  the  narrow  field  follows  the  line  of  sigh  of  the  operator  provided  by  an  eye-tracking 
system.  Over  the  years,  it  became  obvious  to  the  team  that  the  narrow  field  of  view  opto-mechanical 
pointing  system  could  be  used  alone,  without  the  wide  field  of  view  camera  to  produce  a  large  and  high- 
resolution  mosaic  image. 

In  this  paper,  we  will  describe  the  opto-mechanical  pointing  system  used  for  the  step-staring  system  and 
its  different  variations.  We  will  also  describe  the  mathematical  model  and  control  algorithms  used  to  build 
the  mosaic  dynamically  from  multiple  images  by  taking  into  account  the  attitude  and  position  of  the 
airborne  platform.  Moreover,  to  validate  the  step-stare  system  operations  and  the  effectiveness  of  the 
algorithms,  a  simulator  has  been  developed  and  will  be  presented.  This  paper  will  conclude  by  presenting 
preliminary  results  from  the  first  prototype  of  a  high-speed  step-stare  system  and  discussion  of  its  usage  in 
unmanned  platform. 


2.0  STEP-STARE  CONCEPT 

The  step-stare  concept  is  quite  simple.  Using  the  pointing  system  based  on  a  pair  of  achromatic  Risley’s 
prisms  that  were  developed  for  the  Infrared  Eye,  the  step-stare  technique  rapidly  builds  a  high-resolution, 
large  area  coverage  mosaic  from  the  smaller  images.  Building  a  mosaic  in  a  dynamic  context  brings  out 
many  challenges  not  present  in  a  static  context.  Being  on  an  airborne  platform,  the  point  of  view  of  the 
camera  is  changing  constantly,  and  at  a  given  time  instant,  images  that  were  previously  taken  must  be 
warped  to  be  correctly  displayed  with  that  taken  from  the  current  point  of  view.  A  dynamic  strategy  that 
adapts  automatically  to  the  operational  context  was  developed  to  control  image  acquisition  according  to 
criteria  based  on  the  state  of  the  mosaic.  The  images  taken  are  geo-referenced  by  the  projection  of  their 
comer  onto  the  ground  using  flight  data.  They  are  assembled  according  to  the  actual  point  of  view  with 
homo  graphic  transformations. 

The  methods  used  for  display  corrections  due  to  the  airborne  platform  displacement  are  explained  in 
section  4.1.  Also,  since  the  narrow  field  of  view  can  be  positioned  anywhere  within  the  wide  field 
coverage,  we  need  a  strategy  to  select  the  position  where  to  take  the  next  image.  Our  dynamic  acquisition 
strategy  is  described  in  section  4.2.  The  model  used  to  map  pixels  to  geo-referenced  positions  and  vice- 
versa  is  detailed  in  section  3.  We  also  developed  a  software  simulator  to  optimize  and  test  our  algorithms. 
This  simulator  and  some  simulation  results  are  presented  in  section  5.  Section  6  shows  the  prototype  that 
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implements  the  step-stare  strategy  and  includes  a  few  results  from  real  experimentation.  More  details 
about  the  model,  the  strategy  and  the  simulation  are  given  in  Refs.  3  and  4. 


3.0  SYSTEM  MODEL 

We  have  modelled  the  acquisition  system  in  order  to  establish  the  correspondence  between  ground 
coordinates  and  mosaic  pixels.  Therefore  the  model  deals  mostly  with  projections  and  positioning.  The 
main  components  of  the  model  are  a  set  of  prisms,  a  camera  and  an  airborne  platform. 

Two  right-handed  Cartesian  3D  coordinate  systems  are  used  with  the  model:  the  camera  referential  C  and 
the  world  referential  W.  Two  2D  referentials  /  and  M  are  used  to  represent  pixel  coordinates  in  the  camera 
image  and  the  mosaic  planes  respectively.  An  overview  of  the  coordinate  systems  is  shown  in  Fig.  1. 

The  Risley  prisms  are  both  aligned  on  the  optical  axis  of  the  camera.  The  pointing  system  setting  is 
described  in  Ref.  3.  Each  prism  produces  a  deviation  of  the  optical  axis  that  moves  the  narrow  field  image 
centre  away  from  the  mosaic  image  centre.  When  the  two  prisms  are  opposed,  there  is  no  deviation  at  all 
and  when  they  are  aligned,  the  deviation  is  maximal.  The  prisms  can  rotate  and  the  two  deviations  are 
summed  up  vectorially.  In  our  model,  we  represent  the  total  deviation  with  its  horizontal  component  9h 
and  its  vertical  component  9V.  In  Fig.  2,  9h  has  a  positive  value  and  9V  has  a  negative  value. 

The  camera  model  used  in  this  paper  is  the  standard  pinhole  model,  which  performs  a  linear  projective 
mapping  of  the  3D  world  into  the  image  frame.  The  camera  will  be  fixed  under  the  airborne  platform 
using  a  stabilized  pan  and  tilt  mount.  The  optical  axis  of  the  camera  is  oriented  along  the  xc  axis  and  the  zc 
axis  points  towards  the  top  of  the  camera.  The  position  and  altitude  of  the  camera  is  (posx ,  posy ,  alt ) 
relative  to  W.  The  C  and  W  frame  referentials  can  be  aligned  using  a  Roll-Pitch-Yaw  rotation.  The  roll 
angle  y  turns  around  xc,  the  pitch  angle  /?  rotates  around  yc  and  the  yaw  angle  a  makes  the  last  rotation 
around  zc.  These  angles  and  their  direction  of  rotation  (using  the  right  hand  rule)  are  shown  in  Fig.  1 . 


Rotation  matrices  are  used  to  convert  positions  from  one  referential  to  another.  Equation  (1)  is  used  to  find 
the  (xw,  yw,  zw)  coordinates  in  the  world  referential  W  of  the  point  (xc,  yc,  zc)  in  C.  The  reverse  conversion  is 
achieved  by  first  subtracting  (posx ,  posy,  alt )  from  (xw,  yw,  zw)  and  then  applying  the  rotation  matrices  in 
reverse  order  with  negative  angles. 
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ORGANIZATION 


3.1  Ground  Projection 

If  we  know  its  horizontal  y/h  and  vertical  y/v  angles  relative  to  the  image  centre,  we  can  find  the  projection 
on  the  ground  of  any  image  pixel.  These  angles  are  shown  in  Figure  2.  For  the  image  corners,  y/h  =  ±fovih/2 
and  y/v  =  ±foviv/2,  where  foVih  and  foviv  are  the  horizontal  and  vertical  fields  of  view  of  the  narrow  field 
camera  image. 
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For  the  projection,  we  need  a  unit  vector  pointing  in  the  direction  of  the  ground  coordinates  mapped  by  the 
pixel.  We  first  build  a  vector  in  C  pointing  towards  the  desired  pixel  of  the  mosaic  from  a  unit  vector 
oriented  towards  the  optical  axis  (xc).  Two  rotations  are  needed:  a  rotation  of  an  angle  cph  =  -y/h  -  9h  around 
zc  and  a  rotation  around  yc  of  an  angle  cpv  =  y/v  +  0V.  Then,  by  applying  the  camera-to-world  rotation 
matrices  from  Eq.  (1),  we  obtain  the  unit  length  projection  vector  ( xp ,  yp,  zp )  along  W  axes  but  relative  to  C 
origin.  Equation  (2)  combines  these  two  operations. 
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For  the  next  equation,  we  suppose  that  the  ground  is  flat.  Thus  we  need  to  find  the  intersection  of  the 
projection  vector  with  the  xw  -  yw  plane.  We  stretch  it  until  its  zp  component  is  the  same  length  as  the 
aircraft's  altitude  alt ,  but  negative.  The  only  step  left  is  the  translation  from  the  origin  of  coordinate  system 
C  to  W.  This  result  is  given  by  Equation  (3). 
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The  fiat  ground  hypothesis  is  a  good  approximation  if  the  aircraft  flies  over  the  sea  or  at  high  enough 
altitude  over  slightly  uneven  land.  Only  Equation  (3)  depends  on  it.  In  a  situation  where  the  scene  cannot 
be  represented  by  a  plane,  a  3D  model  such  as  an  elevation  map  (like  DTED)  could  be  used  in  order  to 
determine  the  intersection  of  a  specific  projection  vector  with  the  scene  content. 


3.2  Horizon  Limitation 

However,  Equation  (3)  does  not  apply  when  the  projection  reaches  the  ground  too  far  or  points  above  the 
horizon.  We  use  the  visible  horizon  distance  rdist  =  3924.31  \/a/£  (meters),  as  defined  in  Ref.  5  for  this 
specific  case.  If  the  inequality  of  Equation  (4)  is  not  respected,  we  do  not  attempt  to  find  the  ground 
coordinates  because  it  means  that  the  projection  vector  will  not  reach  the  ground  within  the  horizon  limit. 
Instead,  we  will  rather  find  its  intersection  with  a  half-sphere  of  rdist  radius  whose  centre  is  the  {posx ,  posy ) 
position  on  the  ground  under  the  aircraft.  We  stretch  the  vector  (x^,  yp,  zp)  by  a  factor  /  to  reach  the  half¬ 
sphere.  This  intersection  position  will  be  represented  using  the  spherical  coordinate  system  shown  in  Fig. 
3.  Eqs.  (5)  and  (6)  are  used  to  find  the  spherical  coordinates  of  the  intersection. 
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— alt  ^  \A'p  -Jp 
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Centring  the  half-sphere  on  the  aircraft  position  means  that  these  intersection  points  will  travel  according 
to  the  aircraft  displacement.  This  describes  what  would  happen  with  an  object  point  located  at  an  infinite 
distance.  Although  it  is  not  perfectly  accurate,  this  is  not  a  bad  approximation  because  image  points  that 
map  the  sky  and  objects  around  the  horizon  line  can  usually  be  considered  very  far.  With  Equation  (7)  we 
can  find  back  the  Cartesian  coordinates  in  the  W  referential. 

=  r<i,,i  cos(fl)sin(o)  I  po- s, .  yw  -  Tdist  sin(0)sin(<£)  +posy  and  =  /*,/„./  cos (0)  ^ 


3.3  Reverse  Projection 

For  display  purposes,  we  also  need  to  find  the  cph  and  cpv  angles  relative  to  the  mosaic  centre  from  the 
coordinates  (xw,  yw,  zw)  in  frame  referential  W.  After  reversing  the  translation  between  W  and  C,  using 
Equation  (8),  we  normalize  the  vector  to  undo  the  stretching.  Then,  we  make  the  reverse  rotations  of  Eq. 
(1),  which  gives  the  (xc,  yc,  zc)  vector  in  C.  We  know  that  if  we  could  also  revert  the  Rcpn  and  Rcpv  rotations 
of  Equation  (2),  we  would  get  back  the  unit  vector  along  xc.  This  leads  to  the  equality  of  Equation  (9), 
which  is  solved  by  Equation  (10).  The  cos(cpv)  terms  in  Equation  (10)  are  not  simplified  because  they 
allow  us  to  find  the  correct  quadrant  angle. 
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Now  that  we  have  (ph  and  cpv ,  Equation  (11)  gives  the  (xm,  ym)  pixel  position  of  the  object  in  the  mosaic 
referential  M.  We  use  the  fact  that  the  horizontal  and  vertical  fields  of  view  covered  by  the  mosaic,  fovmh 
and  fovmv ,  and  their  corresponding  mosaic  width  wm  and  height  hm  in  pixels  are  known.  As  usual,  the  top 
left  comer  of  the  mosaic  image  is  the  origin  of  the  coordinate  system. 
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ORGANIZATION 


4.0  STEP-STARE  STRATEGY 

As  explained  in  section  2,  the  step-stare  strategy  requires  building  a  mosaic  from  the  images  already  taken 
and  selecting  the  narrow  field  position  from  which  the  next  image  will  be  taken.  More  specifically,  after 
the  initialization  step,  the  operations  of  the  step-stare  approach  are:  take  an  image,  build  the  mosaic  from 
the  images  we  have,  erase  the  obsolete  ones,  evaluate  the  next  position,  move  the  prisms  to  that  position 
and  start  over  again. 


4.1  Mosaic  Display 

The  mosaic  we  generate  is  constructed  with  images  that  were  all  taken  from  different  locations  and 
orientations.  Thus,  we  need  a  reference  to  locate  the  image  position  in  the  real  world.  We  chose  to  use  the 
projection  of  the  four  image  corners  on  the  ground  or  on  the  half-sphere  as  references.  These  positions  are 
recorded  with  each  image  and  can  be  computed  using  the  equations  presented  in  sections  3.1  and  3.2.  For 
these  reference  points,  we  can  find  their  pixel  position  in  the  mosaic  for  the  current  point  of  view,  by  using 
the  reverse  projection  equations  of  section  3.3.  However,  we  cannot  copy  the  image  directly  in  the  mosaic, 
because  its  area  within  the  mosaic  is  generally  not  a  rectangle  due  to  prisms  distortions  and  changing  point 
of  view.  This  is  especially  true  when  the  aircraft  is  turning.  This  situation  is  depicted  in  Fig.  4,  where  wz 
and  hi  are  the  image  width  and  height  in  pixels. 
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To  warp  the  images  into  the  mosaic,  we  use  a  homography  that  associates  each  (w,  v)  destination  pixel  to  a 
(xz,  yt)  source  image  pixel.  The  homography  matrix  is  defined  in  Equation  (12).  We  can  write  a  system  of 
eight  equations  using  the  eight  source  image  corners  coordinates  and  their  associated  destination 
coordinates.  Solving  this  system  will  give  us  the  eight  unknown  matrix  values.  The  equations  are 
simplified  by  the  fact  that  half  of  the  source  image  comers  coordinates  are  zero.  Thus,  we  can  resolve  the 
homography  equation  system  analytically  and  avoid  the  computation  of  a  matrix  inversion  for  each  image. 


Since  the  transformation  is  known,  the  information  about  which  area  an  image  covers  exactly  is  available. 
We  can  take  advantage  of  the  mapping  step  to  compute  statistics  that  will  be  useful  through  the  rest  of  the 
algorithm.  To  keep  track  of  the  mosaic  coverage,  we  use  a  pixel  map.  It  is  a  matrix  that  is  filled  with  the 
number  of  the  image  that  covers  each  pixel.  The  pixel  map  is  ten  times  smaller  than  the  actual  mosaic  plus 
a  half-image  wide  border  whose  use  will  be  explained  in  section  4.2.  After  building  the  mosaic,  we  know 
the  area  in  pixels  an  individual  image  covers.  If  the  image  does  not  cover  enough  pixels,  it  can  be  deleted 
from  the  bank  of  images  and  the  pixel  map  is  updated  accordingly.  The  image  management  is  done  after 
the  mosaic-building  step  but  before  the  next  position  evaluation.  It  allows  anticipating  the  creation  of  a 
hole  in  the  mosaic  (a  set  of  pixels  in  the  mosaic  that  do  not  map  to  any  images)  before  it  actually  appears 
on  the  mosaic.  The  pixel  map  also  makes  it  possible  to  know  which  areas  need  to  be  refreshed  and  the  hole 
locations. 
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4.2  Acquisition  Strategy 

Although  the  mosaic  is  now  ready  to  be  displayed  on  the  operator's  console,  we  still  have  to  decide  where 
the  next  image  will  be  taken.  The  order  in  which  the  images  are  collected  is  significant  because  of  the 
aircraft  motion.  Using  a  set  of  fixed  patterns  requires  that  we  define  every  possible  situation  and  make 
sure  that  an  image-taking  pattern  responds  appropriately  to  it.  The  advantage  of  a  dynamic  acquisition 
strategy  is  that  it  adapts  automatically  to  the  operational  context.  The  aim  of  the  acquisition  strategy  is  to 
make  sure  that  each  pixel  of  the  mosaic  is  covered  and  that  the  refresh  rate  is  as  high  as  possible.  Thus,  the 
strategy  is  dependent  on  the  mosaic  state  only.  The  coverage  and  image  age  information  was  stored  in  the 
pixel  map  through  the  previous  algorithm  steps,  so  we  can  use  it  to  evaluate  the  mosaic  situation. 

The  space  that  the  maximum  prism  deviation  angle  may  reach  forms  a  circle  whose  radius  is  greater  or 
equal  to  the  distance  between  the  mosaic  centre  and  a  comer  image  centre.  The  centre  of  an  image  could 
be  positioned  on  any  pixel  within  this  circle.  However,  an  evaluation  of  all  of  these  positions  would 
require  a  significant  amount  of  computations  to  determine  the  best  pixel  on  which  to  centre  the  next 
image.  In  order  to  reduce  computation  time,  position  selection  is  restricted  to  a  limited  set  of  fixed 
positions.  These  positions  are  laid  in  a  grid  whose  size  is  selectable.  The  mosaic  size  can  be  smaller  than 
the  space  covered  by  the  prisms.  The  positions  aligned  with  the  grid  on  the  mosaic  sides  and  comers  are 
added  to  the  available  step-stare  positions  if  the  prisms  can  reach  them.  These  positions  are  called  outside 
positions  while  the  inside  positions  are  those  on  the  grid  inside  the  mosaic  (Fig.  5). 
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Figure  5:  Inside  and  outside  positions  evaluated. 


Since  we  use  positions  that  are  on  the  sides  of  the  mosaic,  this  creates  a  half-image  wide  border  around  the 
mosaic  that  may  also  be  filled  with  images  (shown  in  dash  lines  in  Fig.  5).  This  is  why  the  pixel  map  is 
larger  than  the  mosaic.  When  we  take  the  border  information  into  account,  we  can  preventively  fill  the 
border  with  images.  This  avoids  seeing  holes  moving  into  the  mosaic  due  to  the  scrolling  effect  induced 
by  aircraft  motion. 

All  positions  are  evaluated  using  these  three  criteria:  minimization  of  prism  displacements,  maximization 
of  hole  coverage  and  refresh  rate.  Each  criterion  is  given  a  normalized  value  that  ranges  between  0  (less 
priority)  and  1  (more  priority).  The  total  priority  score  is  the  weighted  sum  of  the  three  criteria.  The 
operator  can  adjust  the  weights.  The  position  with  the  highest  score  is  the  position  used  to  take  the  next 
image.  To  evaluate  a  given  position,  we  use  the  information  from  a  section  of  the  pixel  map  that  covers  the 
area  an  image  centred  on  this  position  would  fill.  Prism  distortions  are  not  considered  for  this  step  and  this 
section  is  an  image-size  rectangle  centred  on  the  given  position. 

4.2.1  Prism  Displacement 

Minimization  of  the  prisms'  displacement  time  is  desirable.  If  we  spend  less  time  moving  the  prisms,  we 
can  take  more  images  and  improve  the  mosaic  refresh  rate  and  quality. 
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The  total  deviation  of  the  prisms  can  be  represented  on  a  plane  with  polar  coordinates.  To  simplify  the 
equations  we  select  the  plane  parallel  to  the  image  plane  for  which  the  distance  to  the  prisms  is  equal  to  1 . 
For  a  given  ( 6h ,  9V)  deviation,  we  can  find  its  (pt ,  co)  polar  coordinates  with  Equation  (13).  As  shown  in 
Fig.  6,  two  vectors  of  equal  length  p  are  summed  up  to  make  the  total  deviation.  Each  of  these  vectors 
represents  the  projection  on  the  plane  of  the  deviation  caused  by  the  prism.  The  p  length  of  the  two 
vectors  can  be  found  using  the  maximum  deviation  angle  6max  the  prisms  can  reach.  With  Equation  (14), 
we  obtain  the  value  of  6max  and  S.  The  final  prism  angles  are  co  -  S  and  co  +  3. 


pt  —  \/ tan(#h)2  +  tan(#^)2  and  lj  =  arctan(  \ ) 


ttvn(<9/0  (13) 


p  = 


tan (0max )  _  ]  ^ _ _ Pt 


and  S  =  arccosf  —  ) 

2 P  (14) 
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Figure  6:  Individual  prism  angles. 

We  can  find  the  prism  angles  corresponding  to  the  next  image  centre  position  using  the  method  just 
described.  Because  each  of  the  two  prisms  can  move  to  either  of  the  two  destination  angles,  there  are  two 
possible  displacements.  The  total  time  of  the  displacement  is  dependent  only  on  the  prism  with  the  greatest 
rotation  angle.  We  find  this  angle  for  each  possibility  and  then  we  keep  the  possibility  for  which  the 
rotation  angle  is  the  smallest.  The  duration  of  the  displacement  is  not  linearly  related  to  the  angular 
distance  ang  travelled.  To  approximate  this  time,  we  suppose  a  constant  positive  acceleration  a  for  the  first 
half  of  the  distance  (ang! 2)  and  an  equal  negative  acceleration  for  the  other  half.  Because  x/2  =  1/2-  a( t/2)2, 
we  know  the  displacement  time  is  proportional  to  the  square  root  of  the  angular  distance  ang.  The  final 
score  for  this  criterion  is  given  by  '  yjangfn .  The  n  division  normalizes  the  score  and  it  is  subtracted 
from  one  to  give  the  highest  priority  to  short  prism  displacements. 


4.2.2  Hole  Coverage 

Hole  presence  in  the  mosaic  is  an  important  problem  because  there  is  always  a  slight  chance  the  target  was 
missed  because  it  was  located  in  a  small  unfilled  area  of  the  mosaic.  The  hole  score  for  an  image  position 
is  based  on  the  percentage  of  its  area  that  is  not  covered  in  the  pixel  map.  We  consider  holes  inside  the 
mosaic  to  be  more  important  than  those  on  the  border.  However,  filling  the  border  will  avoid  hole 
apparition  inside  the  mosaic.  Therefore,  if  there  are  holes  inside  the  mosaic,  we  consider  holes  on  the 
border,  but  only  if  they  represent  more  than  50%  of  the  position  area.  To  ensure  that  holes  will  obtain  a 
significant  score  in  comparison  with  the  other  criteria,  a  normalization  factor  that  increases  the  priority  of 
the  largest  hole  of  the  mosaic  to  the  maximum  value  is  applied  to  the  all  resulting  scores.  However,  this 
increase  of  hole  importance  means  that  a  very  high  priority  will  also  be  granted  to  a  tiny  hole  if  it  is  the 
largest  of  the  pixel  map.  To  avoid  this  situation  for  border  holes,  we  do  not  normalize  the  score  if  there  are 
no  holes  inside  the  mosaic  and  all  outside  positions  have  at  least  50%  of  their  area  covered. 

4.2.3  Refresh  Rate 

Refreshing  the  mosaic  rapidly  and  regularly  makes  it  more  accurate  and  improves  detection.  We  define  the 
age  of  an  image  as  the  number  milliseconds  that  passed  since  the  image  was  acquired.  To  evaluate  the  age 
of  an  inside  position,  we  compute  the  age  average  of  the  pixels  it  contains.  For  outside  positions,  we  also 
compute  the  average  of  the  pixel  ages.  However,  for  pixels  located  on  the  border  and  in  holes,  their  age  is 
defined  as  the  median  age  of  the  images  still  present  in  the  mosaic.  The  true  age  of  border  pixels  is  not 
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used  in  the  average  because  its  content  is  not  displayed  on  the  screen  and  we  are  not  interested  in  having  a 
high  refresh  rate  for  it.  Finally,  the  resulting  scores  are  normalized  to  ensure  that  they  all  respect  the  [0,  1] 
range. 


5.0  STEP-STARE  SIMULATOR 

We  have  implemented  a  complete  simulator  to  test  and  develop  our  step-stare  strategy.  With  the  simulator, 
new  algorithms  can  be  implemented  and  tested  rapidly.  We  can  also  use  it  to  test  ideas  that  would 
otherwise  require  expensive  settings. 

5.1  Interface 

The  interface  of  the  simulator  is  shown  in  Fig.  7.  It  contains  all  the  controls  needed  to  enter  the  simulation 
parameters  and  command  the  simulation  execution.  Three  images  illustrate  the  state  of  the  simulation. 
First,  the  platform  view  (top-left)  shows  the  aircraft  from  above  and  the  mosaic  and  camera  image 
contours  projected  on  the  ground.  The  ground  image  used  in  the  simulator  is  a  large  IR  image  mosaic  that 
was  built  from  real  flight  data.  Next,  the  mosaic  view  (bottom-left)  shows  the  result  obtained  with  the 
step-stare  algorithm.  Optional  frames  around  the  camera  images  can  be  added  to  help  distinguish  the 
image  positions.  Finally,  the  algorithm  view  (bottom-right)  shows  the  state  of  the  pixel  map,  including  the 
border,  where  the  newest  pixels  are  displayed  with  a  bright  red  and  holes  are  black.  Figure  8  shows  the 
interface  of  the  trajectory  generator  included  in  the  simulator.  It  allows  fast  aircraft  trajectory  creation  with 
the  camera  pointing  to  a  specified  direction  relative  to  the  aircraft  or  constantly  adapting  its  orientation  to 
stare  at  a  target  on  the  ground. 


5.2  Simulation  Results 


The  frames  around  the  images  of  the  mosaics  presented  would  not  normally  be  displayed  on  an  operator 
console.  They  are  added  to  the  images  in  order  to  highlight  where  they  are  inserted  in  the  mosaic  for 
analysis  purposes.  Figure  9  presents  the  order  in  which  the  first  images  are  taken  with  a  5  by  5  image 
positions  grid  when  the  aircraft  is  flying  straight.  A  pattern  of  this  kind  always  appears  in  the  beginning  of 
the  algorithm  since  two  of  the  criteria  favour  the  rapid  covering  of  the  mosaic.  After  that,  if  the  aircraft 
continues  to  fly  straight  with  a  constant  speed,  the  acquisition  pattern  changes  and  becomes  stable.  It  will 


RTO-MP-SET-092 


17-9 


UNCLASSIFIED/UNLIMITED 


UNCLASSIFIED/UNLIMITED 


Step-Stare  Image  Gathering  for  High-Resolution  Targeting 


ORGANIZATION 


mostly  select  images  on  the  sides  that  are  scrolling  away  and  regularly  refresh  some  other  position  within 
the  mosaic  when  they  become  outdated.  When  the  aircraft  turns,  the  acquisition  strategy  will  once  again 
adapt  and  try  to  fill  the  positions  where  holes  appear  from  the  side.  Figure  10  shows  the  importance  of 
image  warping  when  the  aircraft  is  turning. 


Figure  9:  Straight  flight  mosaic. 


Figure  10:  Mosaic  during  a  turn. 


6.0  SYSTEM  DESCRIPTION 

In  the  last  sections,  it  has  been  demonstrated  that  it  is  possible  to  make  a  high-speed,  high-resolution,  large 
field  of  view  infrared  images  by  using  Risley’s  prisms  and  a  simulator  proved  the  efficiency  of  the 
algorithms  used.  As  in  any  engineering  systems,  it  is  only  when  the  “rubber  hits  the  road”  that  you  know  if 
it  will  work  effectively.  To  validate  the  effectiveness  of  our  system,  a  prototype  based  on  the  concept  of 
the  Infrared  Eye  opto-mechanical  pointing  device  has  been  built.  Over  the  last  few  months,  experiments 
have  been  conducted  to  quantify  and  to  improve  the  pointing  accuracy  of  the  prism  device.  In  the  next 
sections,  a  description  of  hardware,  software  and  the  calibration  process  will  be  presented. 

6.1  Hardware  Description 

The  step-stare  system  is  composed  of:  a  Indigo’s  Phoenix  LWIR  camera  (8pm-9.2pm,  snapshot 
integration,  640x512,  digital  output),  a  100  mm  Risley’s  prisms,  driven  by  2  -  150  W  Maxon  brushless 
motor,  a  1  axis  fibre-optic  gyro  coupled  with  a  2-axes  tilt  sensor  and  an  acquisition  computer  (Figure  11). 
The  tilt  sensor  and  fibre-optic  gyro  are  used  together  to  provide  precise  reading  of  rotation  on  the  azimuth 
axis  where  the  tilting  induced  inaccuracy  is  compensated  by  the  tilt  sensor.  It  is  worth  mentioning  that,  in 
the  current  test  phase,  the  step-stare  camera  head  is  kept  stationary  (no  translation);  we  only  induce 
rotation  to  the  platform. 

The  mosaic  creation  needs  a  tight  synchronisation  between  the  components  of  the  system.  First,  the 
acquisition  computer  (AC)  computes  the  next  best  view  as  defined  in  section  4.2  and  sends  a  command  to 
the  motor  drivers.  When  they  reach  their  commanded  positions,  the  AC  takes  a  new  frame  from  the 
camera.  At  the  completion  of  the  camera  integration  time,  the  AC  acquires  the  actual  position  and  attitude 
of  the  camera  and  computes  a  new  position  for  the  prisms.  During  the  prisms  displacement,  the  AC 
processes  the  acquired  images,  as  described  in  section  4.1  and  sends  them  to  the  operator  station. 
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Figure  11:  Prototype  of  the  step-stare  system.  Complete  system  with  the  Phoenix  camera 
(left)  and  the  opto-mechanical  pointing  device  (right). 


6.2  Software  Description 

The  step-stare  control  program  interface  is  similar  to  the  simulator.  They  both  use  the  same  version  of  the 
strategy.  All  hardware  drivers  have  been  wrapped  with  COM  interfaces.  The  control  program  is  set  up 
from  XML  files  and  creates  the  drivers  objects  dynamically  using  this  configuration  information.  It  means 
that,  as  long  as  its  drivers  implement  the  appropriate  interface,  a  hardware  system  can  be  replaced  with 
another  without  recompiling  the  application.  Only  the  configuration  file  has  to  be  updated.  Simulated 
hardware  drivers  have  been  developed  for  the  simulator.  In  our  implementation  of  the  strategy,  Direct3D 
from  Microsoft  carries  out  the  warping  of  source  images  to  their  destination  in  the  mosaic  according  to  the 
homography  matrix  in  real-time  (section  4.1).  Image  acquisition  and  prisms  displacement  commands  are 
issued  asynchronously  in  order  to  optimize  code  execution. 

6.3  Prism  Deviation  Calibration 

In  order  to  accurately  aim  the  NFOW  using  the  Risley’s  prism,  a  procedure  has  been  developed  to 
establish  a  relation  between  angular  position  of  the  prisms  and  the  resulting  NFOV  deviation  angles  in 
elevation  and  azimuth.  As  seen  in  Figure  12,  aiming  of  the  NFOV  is  caused  by  the  vectorial  sum  of  both 
prisms  deviation.  Actually,  the  resulting  deviation  could  be  decomposed  in  two  types  of  prism  motions: 
motions  of  the  prisms  in  which  their  relative  angular  separation  changes  (i.e.  they  turn  in  opposite 
directions)  and  prism  motions  in  which  their  relative  angular  separation  stay  constant  (i.e.  they  turn  at  the 
same  speed).  This  behaviour  is  the  basis  of  the  prism  deviation  calibration  strategy.  For  the  purpose,  the 
step-stare  camera  head  is  mounted  on  a  one-axis  goniometer  and  levelled  in  a  way  that  a  distant  target  (e.i. 
a  100  W  bulb,  farther  than  200  m)  moves  on  a  single  line  on  the  image  returned  by  the  camera  when 
goniometer  angle  is  changed.  The  next  step  consists  in  finding  the  prism  angles  that  combined  together 
will  not  induce  image  deviation.  This  will  be  considered  as  the  centre  of  the  mosaic.  From  this  step  begin 
an  iterative  process  in  which  the  goniometer  is  indented  by  few  degrees  and  by  a  combined  motion  of  the 
prisms,  the  distant  target  is  brought  back  in  the  centre  of  the  NFOV.  The  rotation  angles  of  the  prisms  to 
reach  this  point  are  noted.  This  process  is  repeated  till  the  maximum  deviation  of  the  prisms.  The  relation 
between  the  deviation  angle  and  prisms  angle  are  noted  in  a  lookup  table  that  will  be  used  at  run  time  to 
reach  a  point  in  the  mosaic  image  space  by  the  prisms.  Linear  interpolation  is  used  for  intermediate  value 
without  significant  error.  A  point  in  the  mosaic  image  space  is  then  viewed  as  the  length  of  a  vector  from 
the  centre  of  the  mosaic  to  this  point  (that  could  be  found  in  the  lookup  table)  and  a  rotation  from  the 
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calibration  line  to  the  point  (see  Figure  12).  To  bring  the  NFOV  in  this  position,  just  sum  values  from  the 
table  and  the  rotation  to  bring  the  vector  in  position. 
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Figure  12:  Deviation  of  the  NFOV  image 
VS  prisms  rotation.  If  prisms  rotate  in 
opposite  direction,  deviation  will  be  on 
a  radial  line.  If  prisms  rotate  at  same 
speed  in  same  direction,  deviation  will 
be  on  a  constant  radius.  Combination 
of  motion  could  bring  the  NFOV 
anywhere  in  the  mosaic  image  space. 


Figure  13:  Linear  distortion  induced  by 
the  prisms.  The  nine  grey  squares 
represent  the  10°  NFOV  at  their  maximal 
horizontal,  vertical  and  combined 
deviation  positions.  Arrows  show  the 
amplitude  and  direction  of  the  linear 
distortion.  Typical  distortions  measured 
on  the  corner  images  are  ~7%  in  both 
direction. 


6.4  Image  Calibration  and  Distortion  Corrections 

In  the  previous  section,  we  have  presented  the  procedure  to  calibrate  NFOV  deviation  using  the  prisms. 
This  calibration  is  sometimes  enough  to  use  the  system  as  a  general  surveillance  device  to  produce  good 
quality  imagery.  However,  if  the  end  usage  is  aerial  imagery,  where  ground  targets  should  be  geo- 
referenced,  another  level  of  calibration  is  required  in  order  to  correct  optical  distortions  and  rectify  the 
images.  Standard  techniques  generally  found  in  photogrammetry  could  be  used  to  obtain  intrinsic  (and 
distortion)  parameters  of  the  sensor.  Matlab  add-on  and  commercial  product1  are  currently  under  study. 
However,  those  techniques  assume  an  optical  configuration  that  does  not  change  during  the  imaging 
process.  In  our  case,  the  rotating  prisms  change  significantly  the  optical  configuration  and  should  be  taken 
into  consideration  (see  image  13).  To  use  standard  camera  calibration  process,  calibration  data  must  be 
calculated  for  each  imaging  position  as  shown  in  image  5.  At  run  time,  the  corresponding  intrinsic 
parameters  will  be  fetched  from  a  table  and  used  for  image  correction. 


7.0  DISCUSSIONS  AND  APPLICATION 

The  step-stare  imaging  system  presented  in  this  paper  finds  obvious  applications  in  which  high-resolution, 
wide  field  of  view  imagery  is  required  and  in  which  camera  systems  to  achieve  the  task  are  too  big,  heavy 
or  simply  too  expensive  to  risk  loosing  them  (some  UAVs  crash,  apparently!).  One  interesting  aspect  of 
this  concept  is  that  it  could  be  scaled  and  adapted  to  almost  any  kind  of  IR  camera.  It  becomes  thinkable 
to  use  small  bolometer  camera  sporting  resolution  in  the  160x120  pixel  range  with  small  prisms  steering 
mechanism  to  provide  small  tactical  UAV  with  better  imaging  capability.  Such  a  system  could  offer 


1  We  use  PhotoModeler  calibration  package  from  EOS  Systems  inc.  Vancouver,  Canada,  www.photomodeler.com. 
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800x600  pixel  mosaic  image  every  4  seconds  in  a  package  below  3  kg,  assuming  that  the  image  storage  is 
not  on  board. 

The  Risley’s  prism  opto-mechanical  pointing  device  was  adapted  for  multiple  IR  cameras  of  different  IR 
bands  and  has  proved  itself  in  numerous  flight  and  ground  trials  here  at  DRDC  Valcartier.  The  challenge 
now  will  be  to  miniaturize  the  system  for  an  optical  aperture  in  the  25  mm  range  found  on  micro¬ 
bolometer  camera  and  modify  the  mechanical  device  in  a  way  to  compensate  the  torque  induced  on  the 
flying  platform  by  the  acceleration  of  the  prisms. 


8.0  CONCLUSION 

A  novel  concept  of  imaging  system  using  an  opto-mechanical  pointing  device  based  on  Risley’s  prisms 
has  been  developed.  The  step-stare  system  makes  it  possible  to  solve  the  dichotomy  of  the  need  of  a  wide 
field  of  view  for  detection  and  high  resolution  for  identification  by  bringing  the  detection  capability 
equivalent  of  the  narrow  field  in  the  whole  large  area  mosaic  image.  This  technique  could  eventually 
provide  a  small  UAV  with  high-resolution  imager  that  will  be  lighter,  cheaper  and  more  compact.  The 
algorithms  developed  are  general  enough  to  be  used  with  other  flight  systems  (like  commercial  stabilized 
“ball”)  or  for  ground  surveillance.  One  important  factor  to  evaluate  is  to  what  extent  positional  errors, 
rotational  errors  and  measurement  delays  affect  the  creation  of  the  mosaic  and  what  can  be  tolerated.  If 
errors  are  too  important,  we  will  have  to  consider  adding  features  like  content-based  image  registration. 
Experimentations  with  the  system  installed  on  a  rail  with  carriage  mounted  on  the  roof  of  a  building  to 
simulate  scaled  down  flight  displacements  (6  degrees  of  freedom)  are  planned  for  the  months  to  come. 
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